web stats
 
   
 
Selected Development Project
 
Project Title

New Item Response Theory Models for Rater Errors
偵測評分者錯誤的新試題反應理論模式

 
Principal Investigator Professor WANG Wen Chung
 
Area of Research Project
Psychological Studies
 
Project Period
From 01/2017 To 12/2018
Objectives
  1. To develop new IRT models for centrality and similarity in the frameworks of facets models and hierarchical rater models
  2. To conduct simulation studies to evaluate parameter recovery of the new IRT models and the consequences of ignoring rater errors of centrality and similarity
  3. To demonstrate implications and applications of the new models through a series of empirical examples of rater errors
  4. To create new research lines by conducting pilot studies on extensions of the new models
  
Methods Used
This project aims to fulfil the objectives by inventing new IRT models in the frameworks of facets models and hierarchical rater models that consider not only severity and inconsistency but also centrality and similarity. It will involve simulation and empirical studies to evaluate the new models and demonstrate their applications.
Summary of Findings
  1. simulation studies found the parameters of the new IRT models can be well recovered in the facets models and hierarchical rater models
  2. Ignoring rater errors of centrality and similarity resulted in biased item and person estimates.
  3. An application of the new models to a national-wide exam in Taiwan showed that the new models can effectively measure rater errors of severity, centrality and similarity
Impact
The project will make theoretical and practical contributions to the assessment of rater errors. Theoretically, new IRT models will be developed to fix the deficit, which in turn will generate new research lines. Practically, test scores will become fairer when the major rater errors are considered and fairer scores will lead to fairer and better decisions.
Selected Output

  1. Jin, K.-Y., & Wang, W.-C. (2017, July). A new Rasch facets model for rater’s centrality/extremity response style. Paper presented at the International Meeting of the Psychometric Society 2017, Zurich, Switzerland.

  2. Qiu, X.-L., & Wang, W.-C. (2017, July). New item response theory models for rater errors. Paper presented at the International Meeting of the Psychometric Society 2017, Zurich, Switzerland.

Biography of Principal Investigator

Professor Wang Wen Chung is Chair Professor of Department of Psychology and Co-Director of Assessment Research Centre at The Education University of Hong Kong. He obtained a PhD degree from University of California at Berkeley. He was installed in the 2014 Class of the American Educational Research Association (AERA) Fellows. The AERA Fellows Program honors education researchers who have substantial research accomplishments, conveys the association’s commitment to excellence in research and emphasizes the importance of sustaining excellent research in the field.
His research interests included Rasch measurement, item response theory, computerized adaptive testing, cognitive diagnosis modelling, multilevel models, educational and psychological measurement, and psychometrics.
Funding Source

General Research Fund